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Abstract 


In this paper we provide the first rigorous evidence of the impact of a partnership between public 
middle schools and informal science institutions (ISIs), such as museums and zoos, on student 
outcomes. This study focuses on Urban Advantage (UA), a program in New York City (NYC) 
that explicitly draws upon the expertise and resources of the city’s ISIs, bringing these 
institutions together with NYC public schools to improve science education through intensive 
professional development, access to ISIs for teachers and students, and other science resources. 
We conclude that attending a UA school in eighth grade increases middle school science 
achievement, and there is some evidence that it may also increase the likelihood of passing 
standardized science exams in high school. 

Keywords: science education, informal science institutions, urban education, professional 
development 


I. Introduction 


Students in urban school districts are often disadvantaged in many ways, and researchers 
devote considerable attention to understanding how and why inner-city students underperform 
relative to their suburban and rural peers. Despite the challenges urban settings present, cities can 
also offer unique educational opportunities for students. Among the often-overlooked resources 
available in large cities is the richness of the nonprofit sector, particularly informal science 
institutions (ISIs) such as museums, zoos, and gardens. 

Urban Advantage (UA) is a program in New York City (NYC) that explicitly draws upon 
the expertise and resources of the city’s ISIs, bringing these institutions together with NYC 
public schools to improve science education for students and teachers. UA differs fundamentally 
from traditional museum-to-school collaborations as it provides a hybrid model of formal- 
informal partnerships where the resources of institutions are selected, designed, and shaped 
specifically to align with the science curriculum of NYC’s middle schools. This means that the 
partnership is more intensive than most programs that involve outside resources in schools, with 
a deeper relationship between the informal institutions, the department of education, and the 
schools. Additionally, since the program is designed specifically for the NYC schools, it is more 
closely tied to the science curriculum than typical programs that focus on general science 
enrichment. 

In this paper, we estimate the impact of UA on middle school science achievement as 
well as on high school science outcomes, and our results provide the first rigorous evidence of 
the impact of a partnership between public middle schools and ISIs on student outcomes. In this 
paper, we estimate the impact of UA on middle school science test scores (the eighth-grade 
Intermediate Level Science exam) as well as on early high school outcomes, such as attending a 
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Science, Technology, Engineering, or Mathematics (STEM) high school, and taking and passing 
a science Regents exam in eighth or ninth grade. 1 

Results from the fixed effects models show that, on average, UA students do at least 
0.041 standard deviations better than students at non-UA schools on the Intermediate Level 
Science (ILS) exam. As a falsification test, we estimate the impact of UA on ELA and math 
outcomes, and we find no impact. This indicates that UA has a positive impact on science 
specifically, and that the observed increase in science performance is driven by UA and not 
broader school improvement. Results from linear probability models demonstrate that UA may 
have a small positive impact on passing a science Regents exam in eighth or ninth grade, though 
the effects on taking a science Regents exam and attending a STEM high school and are null or 
slightly negative. 

In the next section we describe the Urban Advantage program. Then we present an 
overview of previous research on the role of nonprofit organizations in improving education 
outcomes, including the role of ISIs in science education. Following we present our 
identification strategy and the data and measures. Finally, we present our results and then our 
conclusions. 

II. Background 

Urban Advantage was launched in September 2004 as a collaboration between eight 
NYC informal science institutions (the American Museum of Natural History, Brooklyn Botanic 
Garden, New York Botanical Garden, New York Hall of Science, Queens Botanical Garden, 
Staten Island Zoological Society, and the Wildlife Conservation Society’s Bronx Zoo and New 
York Aquarium) and the New York City Department of Education (NYCDOE) to provide 
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teachers and students with opportunities to engage in authentic science practice. Grounded in the 
learning goals defined in the New York State Learning Standards for Mathematics, Science, and 
Technology, the UA program is focused on supporting teachers to help their sixth-, seventh-, and 
eighth-grade students carry out long-term scientific investigations, including the “science exit 
projects” that NYC eighth-graders are expected to complete before progressing to ninth grade. 

The program provides 48 hours of professional development for teachers new to UA and 
ten hours each year for teachers continuing in the program. The professional development model 
is designed using an immersion-into-inquiry strategy, which emphasizes providing authentic 
hands-on learning experiences in science for teachers, the nature of scientific work, specific 
science content areas, and the essential features of inquiry in the form of long-term 
investigations. (National Research Council, 2000; Loucks-Horsley & Matsumoto, 2010). After 
choosing a UA partner institution to attend for targeted professional development, teachers learn 
how to plan effective field trips, embed resources in instruction, use UA-provided equipment and 
resources, and teach students the components of experimental design and how to develop 
scientific explanations based on claims, evidence, and reasoning. As part of their training, 
teachers conduct their own scientific investigations, learning first-hand what it means to “do 
science,” which is consistent with the teacher- as -learner model of professional development 
(Thompson & Zeuli, 1999). Additionally, the UA program has developed a variety of classroom 
tools and resources for schools and families to support science learning outside of the classroom. 

In addition to high-quality professional development, teachers also receive monetary and 
material support from UA. For example, Urban Advantage provides science kits with materials 
for UA schools, and it also gives teachers funds to purchase materials for their classrooms. 
Additionally, UA provides vouchers for teachers, administrators, students, and families to take 
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trips to any of the eight partner institutions, and schools also receive vouchers for transportation. 
Schools use these resources in a variety of ways. For example, teachers may take students on 
field trips for free, parent coordinators may organize family field trips on the weekends, and 
teachers can distribute family vouchers to students, which allow the students to take family trips 
independent of the school. 

9 

In the 2011-12 school year, 24% of New York City's public middle schools "across all 
five boroughs (136 schools) participated in UA, and the program served 344 teachers and more 
than 35,000 students. With support from the American Museum of Natural History (AMNH), 
informal science education institutions and school districts in other cities have taken steps to 
implement programs based upon the UA framework in NYC, and a UA program launched in 
Denver in the 2010-11 school year. 

In its early years, UA accepted teachers into the program on a volunteer basis; teachers 
learned about the program from their principals and from other teachers, and those who were 
interested self-selected into UA. Over time, the program has developed a more rigorous protocol 
for accepting both teachers and schools. This is partly due to increased demand and partly due 
to budget reductions, which result from the fiscal constraints experienced by the New York City 
Council and NYCDOE, which fund the program. Rather than expand to provide the program to 
more schools, program staff have opted go deeper within already-participating schools, by 
opening the program to sixth-grade teachers and adding additional teachers per grade. 

Professional development offerings have also evolved over time, as UA staff have 
developed more course offerings for continuing teachers, since participating teachers are likely 
to remain in the program for several years and over time, the balance has shifted from new 
teachers to more continuing teachers. These workshops are open only to teachers who have 
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already participated in the professional development for first-year UA teachers and are designed 
to focus in greater depth on specific content related to the science exit project process and 
provide opportunities for experienced teachers to examine student work and assess students’ 
thinking. To help ensure ongoing participation in the UA program, attendance at continuing 
teacher workshops is required for teachers to continue to receive resources and classroom 
materials provided by the program (Short, Elgendy, Roditi, & Holmes, 2012). 

III. Review of the Literature 

The Role of Nonprofits in Education 

Nonprofits are organizations that use its surplus revenues to provide services and reinvest 
in itself rather than distribute profits or dividends to shareholders. In the U.S., many are 
organized as 501(c)3 tax-exempt organizations and as a whole are one of largest providers of 
social services in the nation. These organizations also have a long history of involvement in 
education. They have provided a variety of educational services such as drop-out prevention, 
tutoring, and mentoring; additionally, nonprofits have often provided literacy, vocational, and 
other programs that supplement what is available in schools. In the past decade, however, these 
organizations have played an increasingly larger and important role working within the education 
sector. In addition to more visibly working within or alongside public school systems, there is 
some evidence that nonprofits have also increased in size and quality (Paarlberg & Gen, 2009). 
Whereas early educational nonprofits were small parent-teacher organizations, many have 
become larger organizations, supported not only by parent dues but also by contributions from 
community and national foundations. Some of these organizations work to support individual 
schools (e.g., parent-teacher organizations), while others work to support entire school systems 
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(e.g., local education funds), and others work to influence policy or train principals or teachers at 
the local, state or national level. 

Nonprofits on the national level include foundations, such as the Bill and Melinda Gates 
Foundation, the Eli and Edythe Broad Foundation, and the Andrew Carnegie Foundation, which 
have provided funding for research, advocacy and service provision at the national, state and 
local level; New Leaders, Teach for America, The New Teacher Project, New Schools Venture 
Fund, the Education Trust, The After-School Corporation, The New York City Leadership 
Academy, and the Annenberg Institute for School Reform, which both provide funding for 
programs and/or are service-providers. These organizations train new and current school leaders, 
bring new teachers into hard-to- staff districts, provide resources for new school initiatives, and 
influence education policy. And many of these efforts have had visible impacts on the field 
(Author, 2004; Xu, Hannaway, & Taylor, 2011; author, 2012;) Nonprofits are also increasingly 
involved in running schools, both through standalone charter schools, charter networks, or 
education management organizations. Nonprofits have also been the largest provider of out-of- 
school time (OST) often running programs at schools or at locations close by, such as a park or 
community center. 

Finally, an important, but little-acknowledged group of education nonprofits that works 
with schools includes museums (e.g., art, natural history), cultural institutions (e.g., orchestras, 
opera houses, theaters), and zoos, aquariums, nature centers, and gardens. Collectively referred 
to as informal education institutions, these nonprofit organizations have a history of working 
with schools as part of their public education mission. In formal science institutions (ISIs), such 
as museums, science centers, gardens, and zoos, are one particular type of informal education 
institution. These institutions were typically founded to provide engaging avenues for the 
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general population to learn about science, and in more recent years they have emphasized the 
importance of science as the world becomes more technologically competitive. Currently, these 
ISIs create a world of rich learning experiences, which include social interaction and active 
exploration for people of all ages (Astor-Jack, Balcerzak, & McCallie, 2010). 

Collaborations between ISIs and Schools 

The traditional relationship between ISIs and schools has been limited, but many 
organizations, such as the National Research Council, National Science Teachers Association, 
National Science Board, Institute of Museum of Library Sciences, and the Center for Informal 
Learning and Schools, are formally recognizing the need for deeper collaboration between 
informal and formal science institutions (schools). Relationships between formal and informal 
science education institutions take various forms all of which seek to combine complementary 
aspects of formal and informal settings to maximize their benefits (Phillips, Finkelstein, & 
Wever-Frerichs, 2007; Adams, Gupta, & DeFelice, 2012). For example, because students attend 
school every day and typically receive science instruction in middle school every day, formal 
institutions are uniquely situated to provide consistent instruction in the framework of an 
organized, sequential curriculum; however, schools also operate under organizational constraints 
(e.g., relatively short class periods, large groups of students, testing requirements) that can make 
it difficult to present authentic, hands-on learning opportunities for students. Informal 
institutions, on the other hand, are designed to create engaging venues for students to do science 
and typically have excellent facilities and resources, but do not often have the benefit of 
structured, repeated visits from the same students. 
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Informal institutions have traditionally been sites for infrequent field trips, which are 
typically initiated by teachers to provide one-time enrichment activities for their students, and 
there has been relatively little continued collaboration between these institutions and schools in a 
systematic way. In recent years, however, the role of ISIs in education has been expanding. For 
example, ISIs are increasingly providing additional resources beyond single-day field trips, such 
as more formalized teacher and student programs. For teachers, ISIs frequently provide teacher 
residency programs, research opportunities, and professional development. These programs vary 
in their intensity, with some leading to official degrees or certification. For example, the 
American Museum of Natural History has its own Master of Arts in Teaching program, and the 
New York Hall of Science offers a state certification program in conjunction with The City 
College of New York (Saxman, Gupta, & Steinberg, 2010). For students, ISIs typically feature 
family outreach programs, camp-ins, activity kits, various activities and materials, and out-of- 
school time programs (Astor-Jack et al., 2010; Hein, 1998; Hofstein & Rosenfeld, 1996; 
Inverness Research Associates, 1996; Ramey-Gassert, Walberg, & Walberg 1994; Kisiel, 2010). 
In the Centre for Informal Learning and Schools’ survey of 345 informal science institutions, 73 
percent reported providing “support in the way of programs, workshops, materials, curricula, etc. 
for districts, schools, teachers, or students in the broad area of science education besides a one- 
day field trip” (Phillips et al., 2007). 

Despite this expanding role of ISIs in the education sector, explicit partnerships between 
ISIs and schools are more the exception than the rule, and most interactions between schools and 
ISIs are informal and primarily dependent on the actions of individual teachers (Kisiel, 2010). 
Additionally, studies suggest that science museum resources are generally underused by the 
teachers and students they hope to serve; 53 percent of the informal institutions responding to the 
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Centre for Informal Learning and Schools’ survey reported that their programs could handle 
more participants than they currently serve, while only 24 percent indicated they turn away 
potential participants due to capacity constraints (Phillips et al., 2007). Research suggests that 
external factors such as rising costs and accountability concerns likely influence teacher and 
school participation (Anderson, Kisiel, & Storksdieck, 2006; DeWitt & Storksdieck, 2008). 

Impact of Science Interventions 

There has been little rigorous research examining the impact of science interventions on 
students’ academic outcomes, and even less on formal-informal partnerships. Most quantitative 
studies of science interventions are descriptive or correlational, and qualitative work has focused 
on case studies. Descriptive studies have shown that gender, socioeconomic status, race, reading 
and math ability, and English language learner (ELL) status are important student-level 
predictors of science test scores (Maerten-Rivera, Myers, Lee, & Penfield, 2010). Evaluations of 
specific interventions tend to focus on small samples of students or use survey data to measure 
the intervention’s success in educating students (as opposed to using test scores or other 
measures of academic achievement). 

Lor example, one study of 404 middle and high school students finds that a joint program 
between a museum and an academic medical system (containing a museum exhibit and 
subsequent workshop) increased student interest in health-related careers (Bunce, Griest, 
Howarth, Beemsterboer, Cameron, & Camey, 2009). Another study of 444 middle school 
students finds that three online game sessions were able to increase students’ science knowledge 
about exposure to toxic chemicals, as measured by pre- and post-tests (Klisch, Miller, Wang, & 
Epstein, 2012). Linally, an evaluation of a teacher- level intervention that provided a three-hour 
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workshop and a kit of materials to middle school teachers for cancer education lessons reports 
that teachers perceived that their students gained knowledge from the unit (Bums, 2012). 

There is little quantitative evidence on the impact of partnerships between schools and 
ISIs on student achievement, but studies suggest these collaborations may be more successful 
when school-based and informal educators are able to work together to provide cross-contextual 
learning experiences in both schools and ISIs. For example, one study concludes that it is 
beneficial when school-based and informal educators work together to plan learning experiences 
and when fonnal educators introduce concepts prior to informal learning, facilitate students’ 
reflection of informal learning experiences, and assess the learning that takes place in informal 
settings (Voss 2011). 

Participation in informal science education has also been found to play a role in students’ 
long-term career decisions, particularly among women and those from minority and low-income 
communities. Several studies find that by engaging students, encouraging authentic inquiry, 
building academic knowledge and skills, developing self-efficacy in science, decreasing external 
barriers and increasing supports, and exposing students to STEM careers, ISIs can make STEM 
careers an appealing and viable career choice (Dorsen, Carlson, & Goodyear, 2006; Darke, 
Clewell, & Sevo, 2002; Fadigan & Hammrich, 2004). 

While these studies are helpful in understanding the implementation and success of 
certain science interventions, none examines student achievement directly, and most use only a 
small sample of students from a case study or small-scale intervention. This study contributes 
significantly to the literature by using rigorous methods and a large sample to estimate the 
impact of UA on an at-scale science intervention, and it is one of the first quantitative studies to 
examine formal-informal collaborations. 
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IV. Data and Measures 


Our analysis draws on a rich student-level longitudinal database for NYC public schools 
and students from 2003-04 to 2009- 10. 4 Every student record contains detailed demographic, 
program, and academic information including birthplace, race, gender, language ability, poverty 
(free/reduced-price lunch status), attendance rates, admit/discharge dates and codes, participation 
in special education and language programs, and standardized test scores. These data are 
combined with publicly available data from the Annual School Reports and State Report Cards , 
prepared annually by the NYCODE and the New York State Education Department (NYSED). 

In addition to the rich detail and breadth of our data, unique student identification numbers allow 
us to track students from their entrance in the NYC public school system until their departure. 
Our sample includes eighth-graders from 2004-2010, for a total of more than 400,000 student- 
year observations. 

We measure short-term outcomes using the eighth-grade Intermediate Level Science 
(ILS) exam. New York State requires that all eighth-grade students take the ILS test, which 
consists of approximately 80 questions in three sections: multiple choice, open-ended, and 
performance-based questions. The test covers three standards: scientific inquiry, living 
environment, and physical setting. We measure student performance on the ILS with a 
standardized score (“z-score”), a measure of relative performance standardized across students 
within a grade to have mean 0 and standard deviation 1. Students performing above (below) 
average relative to other students in their grade have positive (negative) z-scores. 

We use several different measures for long-term outcomes. First, we examine students’ 
academic performance using their likelihood of attending a STEM high school. In New York 
City, many high schools offer multiple specialized academies students can choose from, such as 
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health professions, technology, law, journalism, computer science, humanities, and performing 
arts. Schools vary both in terms of how many specialized tracks (if any) they offer and what 
types of curricula these programs provide. For the purposes of this analysis, we define all-STEM 
schools as those that offer only science-rich academies - that is, all students in the school are in a 
science-specific program. We define partial-STEM schools as those schools that offer both 
science-based academies and non-science academies to students. If UA fosters a greater 
appreciation for and understanding of science, then it is possible that UA students will be either 
more interested in STEM schools, more qualified to attend them, or both. 

Two other long-term outcomes we examine are taking and passing a science Regents 
exam (in our case, Living Environment and Earth Science) in eighth or ninth grade. We use this 
measure because students in New York have some choice about what Regents exams to take as 
well as when to take them . 5 

We measure UA program participation at the school level. Data constraints prevent 
creating a student-teacher match, so we estimate the impact of attending a UA school - not the 
impact of having a UA teacher. That is, we classify a school as being a UA school if at least one 
teacher in the school is in the UA program; thus, different schools will have different 
concentrations of UA teachers within the school. Because it is impossible to retract the treatment, 
we designate a school as being a UA school in every year after it joins the program, even if it is 
not currently participating in the program. Thus, once a school becomes a UA school, it is 
always a UA school for the purposes of our analysis. Note, however, that we only include UA 
schools that are in the program for at least two years, since the program is unlikely to be fully 
implemented until year two . 6 
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V. Models 


We estimate a series of models to assess the impact of attending a UA school on student 
outcomes. When we use the short-term outcome (student’s standardized state science exam 

' I 

score ,), these models are standard fixed effects models. For the long-term outcomes, though, we 

o 

use linear probability models, as the outcomes are dichotomous. Our initial specification models 
student outcomes as a function of only attending a UA school, and then we add year effects, 
student controls, and school effects. This model is as follows: 

( 1 ) Yijt = [lo + Pi UAijt + P2 ST n + Yt + ctj + s ij t 

Here, Y is the outcome of interest for student i in school j in year t; UA is an indicator variable 
that takes a value of one if, in year t, student i attended a school j that was a UA school; ST is a 
vector of student characteristics in year t. Models also include year (y) and school effects (a), 
and s represents the remaining variation. Robust standard errors are appropriately modified to 
reflect clustering of students at the school level. In this specification, Pi represents the impact of 
attending a UA school on student achievement, controlling for other characteristics of the 
student. 

Because it is possible that schools positively select into UA (teachers in higher-performing 
schools choose to join the program), we include an indicator variable to control for performance 
in the year prior to joining UA. Thus, the second specification includes an indicator variable that 
takes a value of one if, in year t, student i attends a school j that is in the year prior to joining 
UA. This model is as follows: 

(2) Yijt = Po + P 1 P vc UAijt + P2UA ijt + P31ST ^ + y t + ctj + s //; 

Here, pi represents the difference between UA schools and non-UA schools in the year prior to 
joining the program, and P2 is the impact of UA on student outcomes. 
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Finally, the third specification distinguishes between the first year a school joins UA and 
all subsequent years (the second year in the program and beyond). This takes into account the 
fact that programs are rarely fully implemented in the first year, as described above. 
Consequently, UA concepts and methods may not be implemented until at least the second year a 
school is in the program. The specification we use to model this relationship is below: 

(3) Y ijt = p 0 + PiPreUA ijt + p 2 BaseYr ijt + p 3 PostYrs ijt + PAT,, + y t + ccj + s ijt 
In this model, BaseYr is an indicator variable that takes a value of one if student i in year t 
attends a school j that is in its first of year of the UA program. PostYrs is an indicator variable 
that takes a value of one if student i in year t attends a school j that has been in UA for at least 
two years. Here, pi represents the difference between UA schools and non-UA schools in the 
year prior to joining the program, p 2 is the impact of UA during the first year a school is in UA, 
and p 3 is the impact of UA in all other years after a school joins UA. 

VI. Results 

Table 1 provides descriptive statistics for the characteristics of the students at UA and 
non-UA schools in 2009- 10 9 . UA schools are, in many respects, quite similar to other New 
York City public schools serving eighth- graders. 10 One consistent difference between UA and 
non-UA schools, however, is size; this is also the only statistically significant difference between 
UA and non-UA schools in 2009-10. 

Across the years in our sample, the average size of a UA school ranges from about 650 
students in 2006-07 to more than 1000 students in 2004-05; this is compared to between 400 and 
800 for non-UA schools. In the 2009-10 academic year, the average enrollment at a UA schools 
was more than 700, compared to less than 600 at non-UA schools. 
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In some years, UA schools also show small differences in terms of the percentage of 
student demographic and academic characteristics. UA schools have a lower percentage of black 
and a higher percentage of Asian students compared to non-UA schools in 2007-08 and 2008-09. 
Additionally, in 2006-07 UA schools have greater percentages of students taking the math and 
science exams, and in 2009-10 UA schools have a greater percentage of their students taking the 
math exam. Finally, in 2007-08 and 2008-09, UA schools outperform non-UA schools on the 
English Language Arts (ELA) and math exams. Only in these years do we see a statistically 
significant difference in passing rates between UA and non-UA schools. 

Across UA schools, as with city schools as a whole, there is wide variation in school 
characteristics. As the large standard deviations show, UA serves schools that vary in size from 
very large to very small, from a student body where all are eligible for free lunch to those where 
only a small proportion are eligible, and from schools where the majority of students are black or 
Hispanic to those with a more balanced mix of student ethnicities. 

Before turning to our empirical models, we first show descriptive evidence that UA 
schools outperform non-UA schools on the eighth- ILS exam. Figure 1 shows passing rates 11 on 
the ILS test from years 2003-04 (one year prior to the inception of UA) through 2009-10. Note 
that in 2003-04, less than 40% of NYC eighth-graders were proficient in science, considerably 
less than the New York state average of 86%. In the first two years of UA, there are no 
significant differences in student performance between UA and non-UA schools. However, in 
the third year differences begin to emerge, with students at UA schools outperforming students at 
non-UA schools on the ILS exam. In 2007, 44.2% of eighth-graders at UA schools pass the 
science exam compared to 40.5% at non-UA schools (a gap of 3.7 percentage points). 
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This finding is consistent with the school improvement literature that argues three years 
is the minimum amount of time needed to see results from interventions (Fullan & Stiegelbauer, 
1991). It is reasonable to expect UA to take several years to develop into an effective program, 
and similarly it may take individual schools several years of UA participation before they 
implement the program effectively. By its third year of implementation, UA had a more 
developed and stronger program that included not only professional development, but also a set 
of developed materials and resources for teachers to use in the classroom. Additionally, teachers 
attend professional development sessions throughout the year in three different cycles, and so it 
may not be possible for teachers to fully implement UA until their second year as UA teachers. 
As teachers have more time to implement what they learn during professional development into 
their classroom practice, we begin to see improved student achievement across UA schools. 

Short-Term Outcome: Eighth-Grade ILS Performance 

Having provided descriptive evidence of the success of UA schools, we now turn to our 
empirical models to estimate the impact of UA on student outcomes. We first use a model that 
estimates the short-term impact of UA on eighth-grade ILS test scores. For these specifications, 
our analytic sample includes all eighth-graders who have scores on the ILS exam in 2003-04 to 
2009-10 (the period corresponding to one year before the start of UA through the last year for 
which test scores are available). 

As seen in Table 2, coefficients on the UA variable are not statistically significant in the 
most basic models, which control for only year effects (Model 1) and then add controls for 
student characteristics (Model 2). When we add school fixed effects, however (Model 3), there is 
a positive impact (0.04**) of attending a UA school; that is, comparing students in the same 
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school (in years the school is and is not a UA school) reveals a positive impact of program 
participation. This result is robust to controlling for performance in the year before joining UA 
(Model 4), and as the coefficient on the PreYr variable is small and insignificant, we do not see 
evidence of positive selection into UA (at least on the basis of performance on the eighth-grade 
ILS). 

As previously described, UA is not likely to be fully implemented in a school until at 
least the second year in which a school has officially been in the program. To model this 
appropriately, we distinguish between the first year a school is in UA and all subsequent years 
(Model 5). We find that UA does not have a statistically significant impact on ILS scores during 
schools’ first year of implementation, but that the impact of UA in subsequent years is positive 
(0.06*). Model 6 augments this specification by including lagged math achievement in the 
model. This is not a true value-added model, as we do not have a lagged science test score (no 
science test is given in seventh grade); because math and science test scores are highly 
correlated, however, the lagged math score proxies for prior STEM performance. In this 
specification, the impact of UA is positive but not statistically significant. Again, we see no 
evidence of positive selection into UA. 

Finally, in Model 7 we include interactions between attending a UA school and student 
characteristics (race, gender, poverty status) to determine if there are differential impacts of UA 
on different student groups. We find that while black students do worse than white students in 
science overall, UA has some impact on reducing the disparity between black students and white 
students. On average, black students at UA schools score .06 standard deviations higher than 
black students at non-UA schools. We find a similar effect for Asian students at UA schools, 
who perform .07 standard deviations higher than Asian students at non-UA schools. We also 
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find, however, that female students at UA schools do significantly worse than their non-UA 
counterparts (0.03 standard deviations lower). We do not find any statistically significantly 
differences in the contribution that UA makes to the science achievement of Hispanic, white, or 
poor students. 12 

Though we see no evidence of positive selection into UA based on ILS test scores in the 
year prior to joining UA, it is possible that schools that are on a trajectory of improvement are 
more likely to choose to participate in UA. If this is the case, there may be reverse causality, with 
test score improvements predicting UA participation. If UA schools are improving overall 
(potentially as a result of whole-school reforms, other programs, or increased teacher 
effectiveness), we would expect to see similar positive “impacts” of UA on ELA and math test 
scores. As a falsification test, we perform the analysis from Model 5 using ELA and math test 
scores (z-scores) as dependent variables (Table 3), and we find UA has no effect on ELA or math 
test scores. This suggests that UA participation is not simply a proxy for overall school 
improvement and provides further support for our estimates of the impact of UA on science 
achievement. 

Long-Term Outcomes: High School Outcomes 

The results from the linear probability models are presented in Tables 4 and 5. Table 4 
provides the full models for the impact of attending a UA school on the probability of attending a 
STEM high school. Note that the sample of students in these models, as with our previous 
models, includes students in eighth grade between 2004 and 2010; we follow these students to 
their ninth-grade schools to measure ninth-grade outcomes (type of school attended and Regents- 
taking and Regents performance). Thus, the long-term outcomes for students who were eighth - 
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graders in 2010 are measured in 201 1. Overall, there does not seem to be a systematic impact of 
attending a UA school on the probability of attending either an all-STEM or a partial-STEM high 
school. The first four columns show results for all-STEM high schools. As seen in columns 1 and 
2, there are no statistically significant differences in the likelihood of attending an all-STEM 
school between students who attended a UA school and those who did not. After controlling for 
the year before joining UA (Model 3), attending a UA school is negatively related to attending 
an all-STEM school (-0.023*), but this effect disappears after controlling for eighth-grade school 
effects (Model 4). Overall, we find no evidence of a systematic impact of attending a UA school 
on attending an all-STEM high school. 

Columns 5-8 show results for partial-STEM high schools, and here again there is no 
consistent impact. Results from Models 1-3 indicate that students who attend UA schools are 
slightly more likely to attend partial-STEM high schools than their non-UA counterparts 
(coefficients between 0.046* and 0.052*). Once we control for eighth-grade school effects, 
however, this result disappears, and the coefficient becomes negative and statistically significant, 
though quite small in magnitude (-0.009**). 

Table 5 summarizes the coefficients on Post-UA for all the high school testing outcome 
variables. In the model that includes the eighth-grade school fixed effects, all coefficients are 
statistically significant, and most are positive. Students who attend UA schools are slightly less 
likely to take the Living Environment and Earth Science Regents exams, but they are more likely 
to pass these Regents exams. 1 ’ We see a positive impact of attending a UA school in the 8 th 
grade on passing the Living Environment and Earth Science Regents at all three passing cut- 
points: 55, 65, and 85. 14 However, it should be noted that the coefficient on PreUA is also 
positive and significant, indicating that UA schools also have a slightly higher percentage of 
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students passing these Regents in the year prior to joining UA. Since the passing score for the 
Regents is now 65, that UA schools are able to increase the percentage of students who are able 
to pass at 65 or higher, even slightly, is an important finding. 

It is not entirely surprising that the effects of UA on long-term outcomes are inconsistent 
as many school-level factors that are not likely to be correlated with UA participation may be 
important to students’ high school choices and testing outcomes. For example, school guidance 
counselors, parent coordinators, and other supports to eighth-grade students and parents can have 
great influence over the high school choice and placement process. Also, many middle schools 
tend to send students to a specific type of high school, and consequently act as de facto feeder 
schools for certain high schools. Thus, students may select into middle schools in part based on 
where they want to attend high school, which makes the middle school to high school transition 
endogenous. 

In terms of Regents-taking behavior, there are several important factors that are not 
included in our models. First, because of the regulations around licensing and who is eligible to 
teach a Regents-level course, not all middle schools can offer science Regents exams to their 
students. This is a structural factor that is not likely to be influenced by UA participation. 15 
Additionally, the high school attended has a role in the taking and passing of the Regents in the 
models, and this is unobserved in these models. 

VII. Conclusion 

Formal and informal institutions contribute differently to students’ science learning in 
part because of structural/organizational differences. Schools are not designed for ongoing, 
authentic science investigations, as they must operate within the constraints of a school setting 
and generally have fewer science-specific resources. As such, it is unrealistic to expect formal 
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education to model itself after informal education; rather, a comprehensive science education is 
achieved through collaborations among many different types of institutions, both formal and 
informal (Adams et al., 2012). Rosser (1997) describes collaborations between in-school and 
out-of-school learning experiences as a “two-pronged approach to learning.” 

Despite the growing consensus that collaboration between formal and informal education 
organizations is an important component of improving science education in the United States, 
there are still relatively few examples of ongoing, intensive collaborations between schools and 
informal science institutions, and there has been little research on the impact of such 
partnerships. This study provides the first estimates of the impact of a formal/informal science 
program on academic achievement and finds that exploiting the urban advantage with 
collaborations between formal and informal education institutions is an effective way to improve 
science education in urban schools. 

In short, we find evidence that UA improves performance in science: student 
performance on the New York State eighth-grade science exam increases with the 
implementation of UA. Our estimated impact of 0.05-0.06 standard deviations reflect small 
improvement in science achievement; for comparison, results from the Tennessee STAR 
(student-teacher achievement ratio) experiment indicate that reducing class size from 22-26 to 
13-17 students increased third-graders science test scores by 0.05 to 0.1 standard deviations 
(Konstantopoulos & Chung, 2009). No change is seen in student performance on ELA or math 
for eighth-grade students, suggesting the impact estimate is not merely reflecting coincident 
overall school improvement. Exploratory subgroup analyses find that the impact is largest for 
black and Asian students and is less successful for girls than boys. Additionally, we find that 
UA has no significant impact on whether a student attends a STEM high school, but there is 
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evidence that students who attend UA schools are slightly less likely to take but are more likely 


to pass a science Regents exam in eighth or ninth grade. 
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Table 1: Mean characteristics of UA and non-UA Schools, 2009-10 



UA 

Not UA 

Enrollment 

717 

594 


(425) 

(347) 

% Black 

33.5 

39.2 


(29.1) 

(29.5) 

% Hispanic 

41.6 

41.3 


(26.3) 

(26.5) 

% Asian 

11.8 

9.3 


(15.9) 

(14.2) 

% White 

13.0 

10.1 


(19.3) 

(17.6) 

% LEP 

12.1 

12.0 


(12.1) 

(13.0) 

% Free lunch 

70.2 

71.0 


(20.0) 

(19.9) 

% Taking reading exam 

92.6 

91.5 


(5.5) 

(7.3) 

% Taking math exam 

94.9 

93.9 


(5.1) 

(7.0) 

% Taking science exam 

90.2 

87.8 


(15.5) 

(19.3) 

% Passing reading exam 

37.9 

36.6 


(20.0) 

(21.5) 

% Passing math exam 

49.9 

47.6 


(21.9) 

(22.6) 

% Passing science exam 

53.0 

51.4 


(22.4) 

(23.5) 

% Manhattan 

20.8 

20.9 


(40.7) 

(40.7) 

% Brooklyn 

30.2 

34.7 


(46.1) 

(47.7) 

% Bronx 

24.2 

24.2 


(43.0) 

(42.9) 

% Queens 

20.1 

18.9 


(40.2) 

(39.2) 

% Staten Island 

4.7 

1.3 


(21.2) 

(11.5) 

N 

149 

297 


Standard deviations in parentheses 

Bold indicates differences are statistically significant at .05 level or less 
% Passing is the percent scoring in levels 3 or 4 
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Table 2: Impact of attending a UA school on eighth-grade ILS exam z-scores; 2004-2010 



Model 1 

Model 2 

Model 3 

Model 4 

Model 5 

Model 6 

Model 7 

UA Any Year 

0.06 

0.01 

0.04** 

0.04* 





(0.05) 

(0.03) 

(0.01) 

(0.02) 




PreYr 




0.02 

0.01 

0.00 

0.01 





(0.02) 

(0.02) 

(0.02) 

(0.02) 

Yr Entered UA 





0.04 

0.03 

0.01 






(0.02) 

(0.02) 

(0.04) 

PostUAYrs 





0.06* 

0.04 

0.030 






(0.03) 

(0.02) 

(0.04) 

Black 


-0.75*** 

-0.40*** 

-0.40*** 

-0 34*** 

-0.19*** 

-0.41*** 



(0.04) 

(0.02) 

(0.02) 

(0.02) 

(0.01) 

(0.02) 

Hispanic 


-0.55*** 

-0 23*** 

-0 23*** 

-0 23*** 

-0.10*** 

-0 24*** 



(0.03) 

(0.02) 

(0.02) 

(0.02) 

(0.01) 

(0.02) 

Asian 


0.09** 

0.16*** 

0.16*** 

0.16*** 

0.01 

0.15*** 



(0.03) 

(0.02) 

(0.02) 

(0.02) 

(0.01) 

(0.02) 

Female 


-0.06*** 

-0 07*** 

-0 07*** 

-0 07*** 

-0.04*** 

-0.06*** 



(0.01) 

(0.01) 

(0.01) 

(0.01) 

(0.01) 

(0.01) 

ELL 


-0.91*** 

-0.81*** 

-0.81*** 

-0.81*** 

-0.40*** 

-0.81*** 



(0.03) 

(0.03) 

(0.03) 

(0.03) 

(0.02) 

(0.03) 

Special Educ 


-0.62*** 

-0.59*** 

-0.59*** 

-0.59*** 

-0.28*** 

-0.59*** 



(0.02) 

(0.01) 

(0.01) 

(0.01) 

(0.01) 

(0.12) 

Poor 


-0.14*** 

-0.09*** 

-0.09*** 

-0.09*** 

-0.05*** 

-0.09*** 



(0.01) 

(0.01) 

(0.01) 

(0.01) 

(0.01) 

(0.01) 

Lag Z Math 






0.56*** 

(0.01) 


UA*White 







0.01 

(0.03) 

UA*Black 







0.06* 

(0.03) 

UA*Hispanic 







0.042 

(0.03) 

UA*Asian 







0.07* 

(0.03) 

UA*Female 







-0.03*** 

(0.01) 

UA*Poor 







0.00 

(0.01) 

Constant 

18.73 

12.65 

41.43*** 

41.42*** 

45.91*** 

31.35*** 

44.95*** 


(11.71) 

(8.15) 

(6.59) 

(6.58) 

(8.097) 

(6.26) 

(8.15) 

YearFE 

YES 

YES 

YES 

YES 

YES 

YES 

YES 

School FE 

NO 

NO 

YES 

YES 

YES 

YES 

YES 

N 

401270 

401270 

401270 

401270 

401270 

401270 

401270 

R-Square 

0.00 

0.24 

0.35 

0.35 

0.35 

0.55 

0.35 


(1) *p<0.05, ** p<0.01, *** p<0.001 

(2) Robust clustered standard errors in parentheses 

(3) Year dummy not shown 

(4) Not shown, analysis with Lagged Z Reading. Results are the same as model 7. 
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Table 3: Impact of attending a UA school on Eighth-Grade ELA and math, 2004-2010 



Math 

ELA 

Yr Prior UA 

0.011 

-0.001 


(0.024) 

(0.017) 

Yr Ent. UA 

0.036 

0.026 


(0.027) 

(0.021) 

Yr Post UA 

0.014 

0.022 


(0.031) 

(0.023) 

Black 

-0.408*** 

-0.375*** 


(0.022) 

(0.023) 

Hispanic 

-0.270*** 

-0.275*** 


(0.020) 

(0.021) 

Asian 

0.407*** 

0.064** 


(0.027) 

(0.022) 

Female 

0.027*** 

0.194*** 


(0.004) 

(0.004) 

Poor 

-0.074*** 

-0.1 19*** 


(0.008) 

(0.008) 

Constant 

21.546* 

4.157 


(8.614) 

(7.134) 

YearFE 

YES 

YES 

School FE 

YES 

YES 

R-Square 

0.33 

0.32 

N 

425820 

409572 


(1) *p<0.05, ** p<0.01, *** p<0.001 

(2) Robust clustered standard errors in parentheses 

(3) Not shown Special Ed and ELL 
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Table 4: Impact of attending a UA school on the probability o 

All STEM 

Model 1 Model 2 Model 3 Model 4 


attending a STEM high school 

Partial STEM 

Model 1 Model 2 Model 3 


Model 4 


Yr Before UA 


-0.016* 


-0.004** 


UA 

-0.008 

-0.016 


(0.010) 

(0.009) 

Black 


-0.061*** 

(0.013) 

Hispanic 


-0.046*** 

(0.012) 

Asian 


0.100*** 

(0.016) 

Female 


-0.014*** 

(0.002) 

Poor 


-0.017*** 

(0.004) 

Spec Ed 


-0.046*** 

(0.004) 

ELL 


-0.072*** 

(0.008) 

Constant 

-13.873*** 

-11.576*** 


(2.614) 

(2.234) 

School FE 

NO 

NO 

R-Square 

0.002 

0.047 

N 

368854 

368854 


(0.006) 

(0.001) 


-0.023* 

-0.002 

0.051* 

(0.011) 

(0.002) 

(0.020) 

-0.062*** 

-0.034*** 


(0.013) 

(0.002) 


-0.046*** 

-0.030*** 


(0.012) 

(0.002) 


0.100*** 

0. 1 19*** 


(0.016) 

(0.002) 


-0.014*** 

-0.015*** 


(0.002) 

(0.001) 


-0.018*** 

-0.016*** 


(0.004) 

(0.001) 


-0.046*** 

-0.037*** 


(0.004) 

(0.001) 


-0.072*** 

-0.061*** 


(0.008) 

(0.001) 


-9.395*** 

-7.804*** 

49.741*** 

(1.982) 

(0.596) 

(5.821) 

NO 

YES 

NO 

0.047 

0.115 

0.011 

368854 

368854 

368854 



0.014 

-0.009*** 


(0.013) 

(0.002) 

0.046* 

0.052* 

-0.009** 

(0.019) 

(0.021) 

(0.003) 

-0.053 

-0.053 

0.003 

(0.028) 

(0.028) 

(0.003) 

-0.094*** 

-0.093*** 

-0.006* 

(0.025) 

(0.025) 

(0.003) 

0.005 

0.005 

0.002 

(0.026) 

(0.026) 

(0.003) 

0.002 

0.002 

0.006*** 

(0.003) 

(0.003) 

(0.001) 

-0.017* 

-0.015* 

0.007*** 

(0.008) 

(0.008) 

(0.002) 

-0.005 

-0.005 

-0.003 

(0.005) 

(0.005) 

(0.003) 

0.015 

0.015 

0.028*** 

(0.009) 

(0.009) 

(0.003) 

51.488*** 

49.591*** 

40.319*** 

(5.724) 

(5.249) 

(0.866) 

NO 

NO 

YES 

0.019 

0.019 

0.130 

368854 

368854 

368854 


1) * p<0.05, ** p<0.01, *** p<0.001 

2) Robust clustered standard errors in parentheses 

3) Poor is equal to those eligible for free and reduced price lunch 

4) N is all students who were present in NYC public schools in 8th and 9th grade 

5) Time dummy not shown 
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Table 5: Impact of attending a UA school on the probability of taking and passing Regents exams in science 

Living Environment Earth Science 



Took Test 

Passed 55 

Passed 65 

Passed 85 

Took Test 

Passed 55 

Passed 65 

Passed 85 

Year prior to UA 

-0.095*** 

0.031*** 

0.022*** 

0.013*** 

-0.089*** 

0.015*** 

0.012** 

0.026*** 


(0.002) 

(0.002) 

(0.002) 

(0.002) 

(0.002) 

(0.004) 

(0.004) 

(0.003) 

Year in UA 

-0.047*** 

0.014*** 

0.009** 

0.012*** 

-0.020*** 

0.011* 

0.013* 

0.010* 


(0.003) 

(0.002) 

(0.003) 

(0.003) 

(0.003) 

(0.005) 

(0.005) 

(0.004) 

Black 

-0.026*** 

-0.055*** 

-0.112*** 

-0.125*** 

-0.045*** 

-0.105*** 

-0.157*** 

-0.109*** 


(0.003) 

(0.002) 

(0.003) 

(0.003) 

(0.003) 

(0.004) 

(0.005) 

(0.004) 

Hispanic 

-0.031*** 

-0.029*** 

-0.073*** 

-0.110*** 

-0.031*** 

-0.051*** 

-0.094*** 

-0.083*** 


(0.003) 

(0.002) 

(0.003) 

(0.003) 

(0.003) 

(0.004) 

(0.005) 

(0.004) 

Asian 

0.050*** 

0.018*** 

0.041*** 

0.093*** 

0.001 

0.034*** 

0.051*** 

0.070*** 


(0.003) 

(0.002) 

(0.002) 

(0.003) 

(0.003) 

(0.003) 

(0.004) 

(0.004) 

Female 

0.040*** 

-0.012*** 

-0.028*** 

-0.014*** 

0.020*** 

-0.032*** 

-0.046*** 

-0.025*** 


(0.001) 

(0.001) 

(0.002) 

(0.001) 

(0.001) 

(0.002) 

(0.003) 

(0.002) 

Poor 

0.010*** 

-0.015*** 

-0.028*** 

-0.036*** 

0.000 

-0.022*** 

-0.036*** 

-0.034*** 


(0.002) 

(0.001) 

(0.002) 

(0.002) 

(0.002) 

(0.003) 

(0.003) 

(0.002) 

Spec Ed 

-0.094*** 

-0.194*** 

-0.276*** 

-0.103*** 

-0.053*** 

-0.231*** 

-0.259*** 

-0.075*** 


(0.003) 

(0.003) 

(0.004) 

(0.002) 

(0.003) 

(0.006) 

(0.006) 

(0.002) 

ELL 

-0.070*** 

-0 171*** 

-0.247*** 

-0.113*** 

-0.050*** 

-0.168*** 

-0.193*** 

-0.070*** 


(0.003) 

(0.003) 

(0.003) 

(0.002) 

(0.003) 

(0.005) 

(0.005) 

(0.003) 

Constant 

-19.348*** 

-4.700*** 

-30.620*** 

-38.809*** 

99.184*** 

-19.676*** 

-46.375*** 

-55.972*** 


(0.928) 

(0.714) 

(0.954) 

(0.883) 

(0.822) 

(1.548) 

(1.785) 

(1.655) 

School FE 

YES 

YES 

YES 

YES 

YES 

YES 

YES 

YES 

r2_a 

0.056 

0.099 

0.150 

0.195 

0.102 

0.124 

0.174 

0.185 

N 

36 

259131 

259131 

259131 

368721 

121259 

121259 

121259 


1) * p<0.05, ** p<0.01, *** p<0.001 

2) Robust clustered standard errors in parentheses 

3 ) Control variables not shown are: Year 
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Figure 1: Unweighted mean achievement, eighth-grade Intermediate Level Science exam 



Note: This figure uses student-level data, and thus shows the percent of all students in UA schools and all students in non-UA 
schools that pass the eighth-grade ILS in each year. 


J In addition to fulfilling credit requirements, high school students in New York State must take and pass a 
certain number of Regents Exams in order to graduate. While the requirements vary by the year of 
entrance, students generally must take and pass one science Regents to obtain a high school diploma. 

There are no mandated years when students are eligible or required to take a specific exam, but they 
typically take the exam at the end of the related course. Typically, students will take the Earth Science or 
Living Environment Regents in the 8 th or 9 th grade. Because the graduation requirements reward passing 
but do not penalize failing, it is in a student’s best interest to take and pass these exams earlier than later. 

9 

Middle schools are defined as all schools with an eighth grade. 

3 UA does not accept all schools or teachers who apply to be in the program. Current UA schools as well 
as schools that are hoping to join the program submit applications to UA, and UA program staff consider 
factors such as how many teachers in the program have been or are planning to participate in UA, how 
frequently schools have been or anticipate being able to use vouchers to attend IS Is. 

4 The database is housed at author and is updated annually with data from New York City Department of 
Education (NYCDOE). 

5 In addition to fulfilling credit requirements, high school students in New York State must take and pass a 
certain number of Regents Exams in order to graduate. While the requirements vary by the year of 
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entrance, students generally must take and pass one science Regents to obtain a high school diploma. 
There are no mandated years when students are eligible or required to take a specific exam, but they 
typically take the exam at the end of the related course. Typically, students will take the Earth Science or 
Living Environment Regents in the 8 th or 9 th grade. Because the graduation requirements reward passing 
but do not penalize failing, it is in a student’s best interest to take and pass these exams earlier than later. 

6 We drop 62 schools from the sample, as these schools are only in UA for one year. 

7 The standardized score, or “z-score,” is a measure of relative performance standardized across students 
within a grade to have mean 0 and standard deviation. 

0 

Long-term outcomes include attending a STEM high school, taking a science Regents exam in eighth or 
ninth grade, and passing a science Regents exam in eighth or ninth grade. Outcomes are described further 
in the Data section. 

9 Results for all years are similar and available from the authors. 

10 New York City schools have a variety of grade span configurations that include grade 8. Some schools 
are traditional middle schools that serve grades 6-8, while others may be K-8 or 6-12. 

1 1 A school’s passing rate is the percentage of eighth-grade students who scored a 3 or 4 on the ILS. 

12 

We also find no differences for special education or LEP students; results are not shown here but are 
available from authors. 

1 T 

This result is not contingent upon taking the exam. That is, the sample of students used for the models 
estimating likelihood of passing the exam includes all students who are in the NYC public schools in both 
eighth and ninth grades, not just those who take the exam in one of these years. 

14 Prior to 2005, students were required to earn a score of 55 or higher to count a Regents exam toward 
earning a Local Diploma, while 65 was the score needed for Regents and Advance Regents Diplomas. 
Now earning a Regents Diploma is the minimum requirement, so 65 is the lowest passing score. 
Achieving an 85 on a Regents exam is used as a cutoff for admission to certain selective colleges and 
universities. 

15 For a student to be eligible to take a specific science Regents exam, the teacher must be certified in that 
content area. That is, only teachers certified in earth science can teach an Earth Science Regents course; 
only teachers certified in biology can teach a Living Environment Regents course. If a middle school 
science teacher is only certified in general science or a general middle school instruction, her students are 
not eligible to take a the Earth Science or Living Environment Regents exam. 
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